Better handling of a bilingual collection of texts

Statistical machine translation models are trained from parallel corpora, which are collections of translated texts. These texts are usually processed using dedicated tools called “sentence aligners”, which output parallel sentence pairs. However, parallel resources are very scarce in certain languages or domains. Alternative solutions have been proposed that extract parallel sentences from the so-called “comparable corpora”, containing texts in different languages sharing similar topics. But comparable corpora can contain document pairs with various degrees of parallelism. For example, in the Wikipedia corpus, many article pairs are actually parallel. We implement a system to extract parallel sentences from comparable corpora, and apply this system on the Wikipedia corpus. We also propose a method that determines whether two documents are parallel. By comparing sentence aligners with our parallel sentence extraction system, we suggest that extracting the parallel document pairs in a comparable corpus and using a sentence aligner on them might help improve the recall.